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ABSTRACT: This paper describes a further validation of the cultural consensus model 
(Romney, Welter, and Batchelder 1986). Informants fust rated their knowledge in one of 
five semantic domains (birds, countries, diseases, fabrics, and flowers) and then free listed 
words from that domain. Informants next (in most cases) reported which of a set of items 
from a domain they could recognize and finally performed one or two structured tasks 
(matching or triads test and ranking) in the domain. Overall, informants who agreed more 
with others in a structured task (i.e., displayed greater cultural competence) free listed more 
words, rated themselves as more knowledgeable, and reported being able to recognize more 
items than informants who agreed less with others. In addition, consensus estimated answer 
keys for structured tasks corresponded closely with objective external standards when they 
were available. The results suggest that tree listing capacity, in particular, might be useful 
as a rapid and preliminary measure of informants' knowledge levels in specific semantic 
domains. 
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Throughout the history of anthropology, definitions of culture have asserted 
that culture is shared. Yet when anthropologists interview informants, they 
frequently find disagreement This intracultural variation challenges anthro- 
pologists to develop principled methods for determining cultural beliefs and 
measuring informants' levels of cultural knowledge. 

In addressing these problems at the conceptual level over the past several 
decades, anthropologists have come to recognize culture sharing as a matter 
of degree. However, it has only been recently that anthropologists have 
devised systematic methods of measuring cultural knowledge and sought 
to validate their explicit characterizations of cultural knowledge. In his 
comprehensive study of Aguaruna classification of manioc plants, Boster 
(1985) showed that the pattern of agreement among informants reflected 
differences in knowledge that would be expected from various social 
structural and individual factors. Boster (1985) assessed individuals* 
knowledge levels by calculating the amount of agreement between each 
informant's responses to a standard set of interview questions and the 
aggregated responses of other informants. (Although most anthropologists 
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are not aware of his work, Kaufman (1946) was perhaps first to discover 
that an individual's agreement with the aggregate indicates knowledge). 

Romney, Weller, and Batchelder (1986) formalized this idea in their 
cultural consensus model (see also Batchelder and Romney 1988; Romney, 
Batchelder, and Weller 1987). They argued that under a few general 
assumptions, informants' knowledge levels, or cultural competences, in a 
semantic domain can be estimated directly from the pattern of interinfor- 
mant agreement for a standard set of systematic interview questions. 
Specifically, cultural competence indexes the degree of an informant's 
concordance with other informants. Cultural consensus analysis also 
provides estimates of the "culturally correct" answers to the interview 
questions based on weighting informants' responses by their competences. 

Researchers have tried to validate this notion of cultural knowledge by 
comparing cultural competence with indices of knowledge that are not based 
on the agreement among informants. Boster (1985) found that informants 
who agreed more with others in naming manioc plants displayed greater 
test-retest reliability than those who agreed less with others. In addition, 
Weller (1984) and Brewer, Romney, and Batchelder (1991) observed that 
more competent informants in paired comparison tasks exhibited greater 
internal consistency than less competent informants. Furthermore, Romney 
et ai. (1986) showed that when objective external standards for the answers 
to interview questions were available (e.g., for class exams), informants' 
cultural competences were strongly related to their scores based on the 
external standards. Similarly, in several studies (Iannucci and Romney 
1993/94; Romney et al 1986, 1987; Romney, Webster, and Batchelder n.d.) 
consensus estimated answers matched closely the external standards. 

In this paper, I describe another attempt to validate the cultural consensus 
model by comparing cultural competence with other measures of knowl- 
edge in semantic domains, including free listing capacity, self-rating of 
knowledge, and self-reported recognition ability. To the extent that cultural 
competence is the most theoretically informed and useful notion of cultural 
knowledge, this paper also examines the appropriateness of using these 
alternate measures for rapid and/or preliminary assessment of informants* 
knowledge levels. As such, this paper responds to Bernard, Pelto, Wemer, 
Boster, Romney, Johnson, Ember, and KasakofTs (1986) call for developing 
systematic procedures of identifying knowledgeable informants. 

Free listing is a standard elicitation procedure in which an informant lists 
all the lexical items in a semantic domain s/he can think of (Weller and 
Romney 1988). The number of words an informant generates during free 
listing,, which I call free listing capacity, has been used as an indicator of 
knowledge by Gatewood (1983, 1984) and suggested as a loose measure 
of knowledge by Borgatti (1990). Informants' free listing capacities suggest 
their levels of familiarity with and fluency in a particular domain, at least 
in a linguistic sense. While free listing from a domain for a reasonable length 



of time (say 5 or 10 mins.) undoubtedly will not exhaust all the items in 
that domain known to the informant (cf. Brown 1923; Johnson, Johnson, 
and Mark 1951), the number of items listed is mostly a function of the 
total supply of items available to that informant and not merely a result 
of that informant's rate of output (Johnson et al. 1951). 

Self-rating of knowledge in a semantic domain and self-reported recog- 
nition ability of particular items in a semantic domain were used as 
indicators of knowledge by Gatewood (1984). Self-rating of knowledge 
would seem to have some face validity since people would be' expected 
to know whether they are expert in a domain. Similarly, it appears reasonable 
that more knowledgeable informants would report being able to recognize 
more items in a domain than less knowledgeable informants. 

In order to validate the cultural consensus model by correlating cultural 
competence with other indicators of knowledge, there must be genuine 
intracultural variation in informants* knowledge with respect to the sys- 
tematic interview questions. If informants do not differ in their underlying 
knowledge levels for a structured task, then no external variable will be 
reliably associated with competence. In this study, I implemented Weller 's 
(1987) simulation procedures to identify those situations where genuine 
intracultural variation in informants' knowledge regarding systematic inter- 
view questions was suspect This paper also compares informants' responses 
to the systematic interview questions with objective external standards 
and examines informant motivation as a possible influence on informants' 
observed knowledge levels. 



METHOD 
Informants 

Informants were undergraduates at the University of California, Irvine and 
all were native speakers of English. They received course credit for par- 
ticipating in the study. Informants completed tasks in groups of 5 to 15 in 
a classroom, seated individually at desks fairly widely spaced apart 

Procedure 

The relationships among the indices of knowledge were investigated in 
five semantic domains which have been frequently studied by cognitive 
anthropologists and cognitive psychologists: birds, countries of the world, 
diseases, fabrics, and flowers. Informants completed tasks in one domain 
only. 

All informants performed tasks in the same sequence. First, informants 
answered questionnaires in which they gave demographic information about 
themselves and rated their knowledge in a given domain. The self-rating 
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question asked informants "in comparison to other college studies, how 
much do you know about [domain name]?** Informants gave their self- 
ratings on a 7 point scale in which "1" was labeled "practically nothing'*, 
"4** was labeled "about average", and *7" was labeled "a great deal.** This 
self-rating question and scale closely match those used by Gatewood (1984). 
Then, on a separate page, informants free listed from a semantic domain. 
The instructions for the free listing task were: "What are all the kinds of 
[domain name]? Please write down the names of all the [domain name] 
you can think of." 1 Informants were given 10 minutes to write their 
responses. 

Subsequent tasks in a domain included a standard set of works. The 
standard sets of words for each domain were taken from those lexical 
items generated in free listing by a different group of approximately 21 
informants. For each domain, high and low salience lists were constructed. 
Items* salience was operationalized here by frequency of mention in free 
lists (cf. Romney and D'Andrade 1964). High salience lists consisted of 
the 21 most frequently mentioned words in a domain and low salience 
lists consisted of words mentioned by at least two but no more than 40% 
of the informants (see the Appendix for these lists). A full 21 -item low 
salience list for fabrics could not be constructed by these criteria. Therefore, 
words from Gatewood *s (1984) fabrics list which were not already included 
in my high salience or incomplete low salience lists but were mentioned 
by at least 5 of his informants were added to make a complete 21 -item 
low salience list For the countries and fabrics domains, additional combined 
lists were created by merging the high and low salience lists for that domain 
(see below for details). 

Thus, after the self-rating and free listing tasks, informants did additional 
tasks based on a standard word list For combined lists, informants provided 
self-reports of their ability to recognize particular items on the list. All 
informants performed one or two "consensus" tasks, i.e., tasks for which 
responses may be analyzed with cultural consensus analysis. For high and 
low salience lists, informants performed a triads test and ranking task. For 
combined lists, informants only performed a matching task. In some cases 
for high and low salience lists, the final task informants did was a self- 
reported recognition task. (The recognition and consensus tasks are 
described in detail below). 

Informants who performed the high salience list tasks were the first 
informants to participate in the study (except for the flowers domain, in 
which informants who performed the low salience list tasks were first). 
All of the various standard lists were created from their free listing 
responses. Some of these **first-stage" informants were not native speakers 
of English, so they were not included in this study. (Non-native English 
speakers might have knowledge in a domain that is not able to be fully 
expressed or measured in English). However, the standard lists for a domain 



were based on the free lists of all "first-stage*' informants, native and non- 
native English speakers alike. 

Informants for high salience list tasks in the birds, countries, diseases, 
and fabrics domains and informants for the low salience list tasks in the 
flowers domain completed the self-rating and free listing tasks one day 
and then returned two days later for the consensus and recognition (in 
some cases) tasks. All other sets of informants performed each task during 
one interview session. The self-rating, free listing, recognition/consensus 
task sequence prevented the recognition and consensus tasks from influ- 
encing (by exposing informants to domain items) the free listing task. 

For the self-reports of recognition ability, informants indicated on a 
individually randomized questionnaire which items they could recognize 
if they were to see (for birds, fabrics, and flowers), smell (for flowers), 
and/or touch (for fabrics) them. In the birds domain, informants gave 
self-reports of recognition ability only for the low salience list For the 
combined countries list, informants indicated on an individually random- 
ized questionnaire which countries they could locate on a map that did 
not show the names of the countries and cities. The self-report measure 
of recognition ability used here, which is based on a standard set of items, 
contrasts with the one used by Gatewood (1983, 1984), which is based 
on a different set of items for each informant (i.e., the items an informant 
free listed). My measure, though, approaches the one Gatewood (1984: 
515-516) suggested as an improvement over his own. 

For the triads tests, informants were presented with 70 sets of three items 
and asked to circle the item most different in meaning from the other two. 
Each triads test was individually randomized and constructed according 
to a lambda-one, balanced, incomplete block design (Burton and Nerlove 
1976). For the countries domain, informants were instructed to judge the 
geographical proximity of the three countries in a triad. There was no 
triad in which the three countries were contiguous. 

For the ranking tasks, informants received an individually randomized 
questionnaire of 21 items and were asked to rank the items from "1** to "21** 
on some dimension, with "1" indicating the most and "21" the least on 
that dimension. The dimensions for the birds, countries, diseases, fabrics, 
and flowers domains were size, population, contagion, coarseness, and 
fragrance, respectively. Previous work has demonstrated that size and con- 
tagion are major components of the semantic structures of the birds and 
diseases domains, respectively (e.g., Boster 1989; D*Andrade 1976). My 
pilot testing with another sample of UC Irvine undergraduates showed that 
coarseness was a primary organizing factor in the semantic structure of 
fabrics. 

For the countries matching task, informants were given maps indicating 
only the borders of countries (Maps On File 1981). Each informant received 
six pages of maps (in one of six balanced orderings) with numbers 
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indicating the 42 countries they were to identify (see the Appendix for which 
countries were numbered on which maps). The 42 countries consisted of 
all the counties in the high and low salience lists, except the U.S.S.R., which 
was replaced with Thailand (the 43rd most frequently mentioned country 
by high salience list informants). Numbers were randomly assigned to 
countries and matching questionnaires listed country names in individu- 
ally randomized orders. Informants were asked to match the country 
numbers on the map to the country labels on the questionnaire. 

For the fabrics matching task, informants were given 4* x 6* wire-ring 
notebooks of index cards which had 4* x V samples of fabrics stapled on 
the cards, one sample per numbered card. Each fabric notebook contained 
27 different fabric samples (in individually randomized orders) and all 
samples of a particular fabric were cut from the same bolt of material. I 
obtained the fabrics from two local fabric stores and only included those 
fabrics which were labelled by the store with the same term informants 
had mentioned. Thus only 27 of the 32 fabrics from the high and low 
salience lists were used. Informants were asked to match the numbers of 
the fabric samples in the notebook with the fabric labels printed on a 
questionnaire. 

Questionnaires for the self-reports of recognition ability, triads 
tests, ranking tasks, and countries matching tasks were produced with 
ANTHROPAC (Borgatti 1992). ANTHROPAC was also used for many of 
the data analyses. 

Informants' motivation to perform well in the tasks was measured 
indirectly. Laura Yang, who at the time of data collection was an under- 
graduate at the University of California, Irvine, unobtrusively recorded 
the time at which informants for the combined lists arrived to participate 
in the study. All informants in a particular interview session had signed 
up several days earlier to participate in that session. Rosenthal, Kohn, 
Creenfield, and Carota (1966) showed that subjects who arrived at a 
psychology experiment earlier displayed a greater need for social approval 
than those subjects who arrived later. Persons with a high need for social 
approval might be more motivated to perform optimally in a study such 
as this. Thus, arrival time was used as a crude indicator of informant 
motivation. 



RESULTS 

Indicators of knowledge for high and low salience lists 

Tables IA and IB show the summary statistics for the indicators of knowl- 
edge for the high and low salience lists in all domains. The free listing 
measure represents the number of different items an informant mentioned 
in 10 minutes. In making these counts, I did not omit any free listed items 
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TABLE IA 

Descriptive statistics for indicators of knowledge: high salience lists. 



Variable 


Birds 


Countries 


Diseases 


Fabrics 


Flowers 



# informants 


13 


14 


10 


13 


27 


Triads eigenvalue ratio 


7.22 


23.83 


6.39 


6.99 


2.93 


Triads competence 


0.52 


0.76 


0.62 


0.52 


0.32 


Mean (s.d.) 


(0.12) 


(0.12) 


(0.13) 


(0.13) 


(0.12)* 


Range of simulated tri. 












comp. s.d. ? s 


0.06-0.11 


0.05-0.09 


0.06-0.09 


0,06-0.11 


0.O9-0.12 


Ranking eigenvalue ratio 


36.85 


7.87 


6.44 


8.00 


4.17 


Ranking competence 


0.94 


0.79 


0.79 


0.76 


0.61 


Mean (s.d.) 


(0.04)* 


(0.10)* 


(0.26) 


(0.10)* 


(0.15)* 


Range of simulated rank. 












comp. s.d.'s 


0.02-0.06 


0.07—0.15 


n n<_ft i a 




ft tnfl OA 


Free listing 


25.00 


44.29 


19.80 


L3.00 


14.81 


Mean (s.d.) 


(8.52) 


(12.38) 


(6.41) 


(4.88) 


(5.19) 


Self-rating 


3.08 


4.14 


3.90 


3.23 


3.33 


Mean (s.d.) 


(1.38) 


(0.77) 


(1.37) 


(1.30) 


(1.21) 


Recognition 








16.85 


11.78 


Mean (s.d.) 








(2.41) 


(3.93) 



* Genuine intracultural variation in informants' knowledge on this task is suspect 



TABLE IB 

Descriptive statistics for indicators of knowledge: low salience lists. 



Variable 


Birds 


Countries 


Diseases 


Fabrics 


Flowers 


# informants 


23 


26 


26 


27 


13 


Triads eigenvalue ratio 


10.00 


9.67 


4.79 


5.13 


2.15 


Triads competence 


0.58 


0.56 


0.44 


0.42 


0.30 


Mean (s.d.) 


(0.15) 


(0.21) 


(0.13)* 


(0.16) 


(0.18) 


Range of simulated tri. 










0.09-0.16 


comp. s.d.'s 


0.07-0.11 


0.06-0.11 


0.08-0.14 


0.09-0.11 


Ranking eigenvalue ratio 


20.19 


2.15 


9.65 


2.37 


1.94 


Ranking competence 


0.87 


0.43 


0.81 


0.59 


0.47 


Mean (s.d.) 


(0.09)* 


(0.31) 


(0.11)* 


(0.16)* 


(0.20)* 


Range of simulated rank. 












comp. 3,d.'s 


0.05-0.09 


0.15-0.26 


0.05-0.11 


0.09-0.21 


0.11-0.31 


Free listing 


26.61 


46.69 


18.27 


15.70 


18.62 


Mean (sd.) 


(6.91) 


(10.99) 


(6.02) 


(5.86) 


(9.54) 


Self-rating 


3.00 


4.00 


4.12 


3.15 


3.38 


Mean (s4.) 


(1-17) 


(1.02) 


(1.11) 


(1.35) 


(1.19) 


Recognition 


14.74 






13.22 


9.69 


Mean (s.d.) 


(3.32) 






(4.21) 


(5.85) 



* Genuine intracultural variation in informants' knowledge on this task is suspect. 
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(except repetitions) from any informant's free list because I wanted to mimic 
the situation in which a researcher knows nothing or little about the domain 
under study. Less than 5% of informants had missing data on one or a 
very few triads for the high and low salience lists; these missing data were 
replaced with random data. Self-reported recognition ability refers to the 
number of items on a list which an informant indicated s/he could recog- 
nize. 

For cultural consensus analysis, triads data were treated as multiple 
choice data. Although triads data technically violate the local indepen- 
dence (because each item appears in 10 different triads) and homogeneity 
of items (because some triadic judgments are clearly easier than others) 
axioms of the formal process model described by Romney ex al. (1986), 
for my purposes these violations are relatively unimportant. I calculated 
the association between cultural competence and Welter's (1984) data-based 
reliability measure for the triads data in 17 domains from Romney, Brewer, 
and Batchelder's (1993) study: in every case, the two measures correlated 
above 0.99. Therefore, the choice of the process model competence measure 
over the data-based reliability index makes little difference in measuring 
informants* relative knowledge levels. Other researchers have analyzed 
triads data with consensus analysis in similar ways (e.g., Boster 1991). 
The ranking data were analyzed with the ordinal/interval scale data model 
for consensus analysis (Romney et ah 1987) 

Cultural consensus analysis involves factoring the interinformant agree- 
ment matrix (corrected for guessing for multiple choice data) with minimum 
residual factor analysis (maximum likelihood factor analysis for ordinal/ 
interval scale data). If this procedure yields a single factor solution (i.e., 
the first factor's eigenvalue is several (approximately 3) times larger 
than the second factor's eigenvalue), then the agreement data fit the 
consensus model. Informants' loadings on the first factor represent their 
cultural competences, or amount of agreement with others for their responses 
to the systematic interview questions. For the data to fit the cultural 
consensus model, all informants' competences should also be positive except 
for sampling variation. Thus, if the data fit the cultural consensus model, 
informants share knowledge on the culturally correct answers to the 
systematic interview questions. 

For triads tests with high and low salience lists in the birds, countries, 
diseases, and fabrics domains, there were large eigenvalue ratios. The 
eigenvalue ratios for the high and low salience lists in the flowers domain 
were smaller (2.93 and 2.15, respectively), indicating poorer fits to the 
consensus model. There was one informant each with a negative compe- 
tence for the low salience lists in the fabrics and flowers domains. Hence, 
overall, the triads data displayed acceptable fits to the cultural consensus 
model. 

For the high salience lists, ranking data fit the cultural consensus model 
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well in all domains, with large eigenvalue ratios and no informants with 
negative competences. The eigenvalue ratios for the low salience lists in the 
countries, fabrics, and flowers domains were low, signifying poor fits to 
the consensus model. Three informants registered negative competences 
in the low salience list for countries* confirming the lack of fit to the 
consensus model for this list. Thus, the ranking data also fit the consensus 
model in most cases. 

The samples of informants for the high and low salience lists in a domain 
were very similar in terms of their free listing capacities and self-ratings 
of knowledge. Informants for the high and low salience lists in the fabrics 
domain also were comparable to Gatewood's (1984) informants in terms 
of free listing and self-rating: Gatewood's (1984) informants mentioned a 
mean of 11.9 fabric terms and had mean self-rating of 3.06, which are 
very close to the mean values reported in Tables IA, IB, and IV. In addition, 
informants generally displayed greater consensus on triads and ranking tasks 
(as indexed by mean competence) for high salience lists than for low 
salience lists. Informants' self-reported recognition ability was also greater 
for high salience lists than for low salience lists in the fabrics and flowers 
domains. 

To test for genuine intracultural variation in informants' knowledge for 
consensus tasks, I simulated triads and ranking response data for high and 
low salience lists in every domain using Weller's (1987) procedures. I 
simulated triads data by probabilistically creating responses to 70 3-option 
multiple choice questions for a set of simulated "informants." The number 
of informants simulated was equal to the number of informants who had 
performed triads tests for a given list in a domain. Each simulated infor- 
mant was assigned the same probability of "knowing" the culturally correct 
answer to each question, which was the mean observed competence for 
the actual informants who performed that task. A simulated informant's 
response to each question was determined probabilistically and if the 
simulated informant did not "know" the culturally correct answer to the 
question, the response was determined randomly from the 3 response options 
(i.e., simulated "guessing"). The procedures for simulating the ranking 
data are somewhat more complex, although the basic ideas are similar to 
those just described. Interested readers should consult Weller (1987) for a 
thorough description of the simulation procedures. 2 

Thus, by employing these procedures, simulated data sets appropriate for 
consensus analysis were produced. I created 10 simulated data sets for 
each high and low salience list consensus task in every domain, and then 
submitted each simulated data set to cultural consensus analysis. I noted the 
standard deviation of simulated informants' competences for each simulated 
data set This variation in simulated informants' competences results from 
random "guessing" and the probabilistic nature of which simulated infor- 
mants "know" the correct answers to which questions, even though all 
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simulated informants had the same underlying knowledge level. Tables 
IA and IB report the range of these standard deviations of competence for 
the 10 simulated data sets for each high and low salience list consensus task 
in the five domains. If the standard deviation of real informants' observed 
competences fell within this range, then I considered genuine intracultural 
variation in informants* knowledge for this task to be suspect. In such 
cases, the observed individual differences in competence could well be 
due to guessing and sampling variability. As a result, other indicators of 
knowledge might not reliably correlate with competence in these situa- 
tions. 

For the triads tests with high salience lists, there seemed to be genuine 
intracultural variation among informants in 4 of the 5 domains. However, 
for ranking tasks with high salience lists, genuine intracultural variation 
among informants appeared to be present in only 1 of 5 domains. Similarly, 
there was apparently genuine intracultural variation among informants for 
triads tests with low salience lists in 4 of 5 domains, and for ranking tasks 
with low salience lists in 1 of 5 domains. 

Tables IIA and IIB present the correlations among the indicators of 
knowledge for high and low salience lists. Table III shows a summary of 
these correlations, including the mean Pearsonian correlation and cumula- 
tive Z-score for each pair of indicators. Mean Pearsonian correlations were 
obtained from Fisher's (1948) ^-transformations and weighting by sample 
size, and cumulative Z-scores were computed by Stouffcr's method of 
aggregation (Mosteller and Bush 1954) for the z-transformed correlations. 
I inspected scatterplots for all pairs of indicators and found no obvious 
nonlinear patterns or tendencies toward heteroschedasticity. 

These tables demonstrate modest to moderate positive associations 
between cultural competence and the other indicators of knowledge. For 
correlations involving triads competence, the coefficients tended to be larger 
when there appeared to be genuine intracultural variation in knowledge 
for the triads test and when the data fit the consensus model (i.e., excluding 
low salience flowers). In those cases* free listing capacity was most strongly 
related to triads competence, with a mean correlation of 0.44, while self- 
rating of knowledge and self-reported recognition ability were more weakly 
associated with triads competence, with mean r's of 0.21 and 0.28, respec- 
tively. All triads competence x free listing capacity correlations were 
positive for those cases where there was genuine intracultural variation in 
knowledge for the triads tests and where the triads data fit the consensus 
model. The triads competence x self-rating of knowledge correlations, in 
contrast, were negative in 3 of these 7 cases. Even though genuine intra- 
cultural variation in the ranking tasks appeared suspect, real individual 
differences in knowledge for these tasks still may have been present, as 
indicated by the 10 all positive triads competence x ranking competence 
correlations and mean r of 0,29. 



TABLE HA 

Correlations among indicators of knowledge: high salience lists. 



Variables 


Birds 


Countries 


Diseases 


Fabrics 


Flowers 


Triads comp. x ranking comp. 


0,13 


0.06 


0.02 


0.44 


0.25 


Triads comp. x free listing 


0.02 


0.70 


0.63 




0.11 


Triads comp. x self-rating 




0,48 


0.22 


-0.09 


-0.05 


Triads comp. x recognition 










-0.09 


Ranking comp. x free listing 


0.29 


-0.09 


-0.06 


0.36 


-0.02 


Ranking comp. x self-rating 


-0.31 


0.22 


-0.18 


-0.16 


0.03 


Ranking comp. x recognition 








0.36 


-0.31 


Free listing x self-rating 


-Q.05 


0.58 


0.14 


0.09 


0.45 


Free listing x recognition 








0.77 


0.73 


Self-rating x recognition 








0.33 


0.38 



Note: Correlations which are not underlined involve a competence variable from a 
consensus task where genuine intracultural variation in informants* knowledge is suspect 
or the data do not fit the consensus model. 



TABLE IIB 

Correlations among indicators of knowledge: low salience lists. 



Variables 


Birds 


Countries 


Diseases 


Fabrics 


Flowers 


Triads comp. x ranking comp. 


0.48 


0.06 


0.22 


0.52 


0.45 


Triads comp. x free listing 


0.11 


0.67 


-0.08 


0.48 


-0.18 


Triads comp. x self-rating 


-0.05 


0.53 


0.19 


0.20 


-O.08 


Triads comp. x recognition 


0.09 






0.40 


-0.27 


Ranking comp. x free listing 


0.59 


0.00 


-0.11 


0.46 


-0.03 


Ranking comp. x self-rating 


0.37 


-0.08 


-0.06 


0.48 


0.06 


Ranking comp. x recognition 


0.44 






0.60 


0.36 


Free listing x self-rating 


0.40 


0.55 


0.66 


<L5J 


0.43 


Free listing x recognition 


0.56 






0.85 


071 


Self-rating x recognition 


0.56 






0.58 


0.57 



Note: Correlations which are not underlined involve a competence variable from a 
consensus task where genuine intracultural variation in informants' knowledge is suspect 
or the data do not fit the consensus model. 



The correlations between ranking competence and the other indicators 
also showed that informants' ranking competences may have carried some 
small "signal" representing individual differences in knowledge beyond 
the "noise" of guessing and sampling variation. Ranking competence cor- 
related mildly with free listing capacity and self-reported recognition ability 
and very weakly with self-rating of knowledge. Free listing capacity tended 
to be moderately positively associated with self-rating of knowledge, 
although these two indicators were slightly negatively correlated in the birds 
domain for the high salience list The strongest correlations among any of 
the indicators of knowledge tended to be between free listing capacity and 
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TABLE m 

Summary of correlations among indicators of knowledge for high and low salience lists. 



Variables 


Mean r* 


Cumulative Z* 




0.29 


3.63 


Triads comp. x free listing 






All 10 lists 


0.29 


3.77 


7 lists w/ genuine intracult. var, in triads knowl. 


0.44 


4.67 


Triads comp. x self-rating 






All 10 lists 


0.15 


1.72 


7 lists w/ genuine intracult. var. in triads knowl. 


0.21 


1.89 


Triads comp. x recognition 






All 5 lists w/ recog. data 


0.12 


1.05 


3 lists w/ genuine intracult. var. in triads knowl. 


0.28 


2.12 


Ranking comp. x free listing 


0.17 


2.03 


Ranking comp. x self-rating 


0.09 


0.85 


Ranking comp. x recognition 


0.29 


2.83 


Free listing x self-rating 


0.46 


5.76 


Free listing x recognition 


0.74 


8.75 


Self-rating x recognition 


0.50 


5.01 



* Calculated using Fisher's (1948) z-transfonnations. 

b Calculated using Stouffer's method of aggregation (Mostellcr and Bush 1954). 



self-reported recognition ability, mean r - 0.74. Self-rating of knowledge 
and self-reported recognition ability were less strongly related, mean 
r - 0.50. The pairwise correlations between indicators of knowledge were 
not noticeably different in magnitude between high and low salience lists. 

Indicators of knowledge for combined lists 

For the combined lists in the countries and fabrics domains, the mean 
number of items informants free listed and informants' mean self-rating 
of knowledge were very close to the corresponding mean values in those 
domains for the high and low salience lists (see Table IV). Informants' 
matching data were submitted to the multiple choice model of cultural 
consensus analysis. Since a formal model for the matching response format 
has not been entirely worked out (Batchelder and Romney 1989), I used 
the multiple choice model as a good approximation. In these analyses, the 
number of multiple choice options for a given question (i.e., particular 
country or fabric name to be matched) was equal to the number of items 
on the combined list. There were clear single factor solutions, with large 
eigenvalue ratios and no negative competences, for both countries and 
fabrics domains (see Table IV). Thus, the matching data fit the consensus 
model. 

Simulations of matching data also involved the multiple choice format 
The same simulation procedures described earlier for multiple choice data 
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TABLE IV 

Descriptive statistics for indicators of knowledge: Combined lists. 



Variable Countries Fabrics 



# informants 


21 


23 


Matching eigenvalue ratio 


15.02 


12.03 


Matching competence mean (s.d.) 


0.67 (0.21) 


0.60 (0.17) 


Range of simulated competence s.d.'s 


0.06-0.10 


0.08-0.11 


Free listing mean (s.d.) 


47.95 (11.58) 


15.65 (4.90) 


Self-rating mean (s,d.) 


4.14 (0.56) 


3.09 (1.21) 


Recognition mean (s.d.) 


23.57 (8.21) 


17.83(4.31) 



were used here, except that the number of response options was equal to 
the number of items on the combined list The standard deviation of infor- 
mants* observed matching competence was greater than standard deviations 
of simulated informants* competences for both countries and fabrics 
domains (see Table IV). These results signified there was genuine intra- 
cultural variation among informants for the matching tasks. 

Table V shows the intercorrelations among the four indicators for the 
combined lists. In almost every case, matching competence was moder- 
ately to strongly related to the other indicators of knowledge. Matching 
competence correlated most strongly with self-reported recognition ability 
in both domains. In fact, the correlation between these two indicators was 
the highest inter-indicator correlation in each domain. While the associa- 
tion between matching competence and free listing capacity was substantial 
in both domains (r's - 0.68 and 0.44 for countries and fabrics, respectively), 
matching competence and self-rating of knowledge only exhibited a positive 
relationship in the countries domain. As with the high and low salience lists, 
free listing capacity and self-rating of knowledge were not reliably related 
in the combined lists. Also similar to the high and low salience lists, self- 
reported recognition ability tended to be more strongly correlated with 
free listing capacity than with self-rating of knowledge. 

table v 



Correlations among indicators of knowledge: Combined lists. 


Variables 


Countries 


Fabrics 


Competence x free listing 


0.68 


0.44 


Competence x self-rating 


0.49 


-0.04 


Competence x recognition 


0.85 


0.67 


Free listing x self-rating 


0.37 


0.04 


Free listing x recognition 


0.60 


0.61 


Self-rating x recognition 


0.60 


0-27 
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Time of arrival at the interview session was not strongly or consistently 
correlated with any indicator of knowledge (the r's involving arrival time 
ranged between -0.24 and 0.35 for the two domains). In particular, arrival 
time (large negative values indicated early arrival) was associated with 
matching competence r - -0.18 in the countries domain and r - 0.04 in 
the fabrics domain. 

Agreement among indicators of knowledge 

To test which indicators agreed most with other indicators in measuring 
informants* knowledge levels, I performed minimum residual factor analyses 
of two inter-indicator correlation matrices. If the analyses showed single j 
factor solutions, this would indicate that the different indicators were 
converging in their measurement of a single underlying knowledge con- 
struct. Indicators' loadings on the first factor in such a scenario would 
represent their relative efficacy in measuring that construct. The first inter- 
indicator correlation matrix was assembled from the correlations in Table 
HI, excluding ranking competence and including the correlations with triads 
competence based on those cases where genuine intracultural variation 
was present The second correlation matrix included mean correlations for 
the pairs of indicators listed in Table V. 

Factor analyses showed single factor solutions for both matrices, with 
eigenvalue ratios of 11.9 and 10.2, respectively. For the first correlation 
matrix, the loadings on die first factor were 0.41, 0.91, 0.55, and 0.82 for 
triads competence, free listing capacity, self-rating of knowledge, and self- 
reported recognition ability, respectively. The somewhat low loading for 
triads competence most probably was due to the restricted range of intra- 
cultural variation among informants for the triads tests. For the second 
correlation matrix, the loadings were 0.79, 0.65, 0,37, and 0.99 for matching 
competence, free listing capacity, self-rating of knowledge, and self-reported 
recognition ability, respectively. In both analyses, the self-rating of knowl- 
edge loadings were the smallest of the alternate indicators. 

Validation of consensus model with objective external standards 

I compared the consensus estimated answer keys (based on a procedure 
that weights informants' responses by their competences - see Romney et 
al 1986, 1987 for details) with objective external standards (when avail- 
able) to test further the validity of the cultural consensus model. For the 
high salience list of countries, the consensus estimated rank order correlated 
0.71 with the rank order of the countries' actual populations (1990/1991 
estimates obtained from Hoffman 1993 and State Committee of the U.S.S.R. 
on Statistics 1989), while the answer key based on the rank order of 
countries' mean ranks correlated 0.72 with the rank order of the actual 
populations. For the low salience list of countries, the consensus estimated 



rank order correlated with the rank order of the actual populations 0.84, 
whereas the mean rank answer key correlated with the rank order of the 
actual populations 0.85. That the consensus and mean rank answer keys 
did not really differ in their correspondence to the external standard is 
consistent with the earlier finding that genuine intracultural variation among 
informants for these ranking tasks was suspect. Weighting informants' 
responses by their competences would not improve the validity of the answer 
key if the variance in competence was merely the result of guessing and 
sampling variation. 

For the countries matching task, the consensus estimated answer key 
showed all correct answers according to a world atlas (Rand McNally and 
Co. 1987), while the majority response answer key yielded one incorrect 
answer according to the atlas. In the fabrics domain, 3 items on the 
consensus estimated answer key were incorrectly matched according to 
the fabric labels from the local fabric stores. Four items, however, were 
incorrectly matched (with respect to the fabric stores* labels) on the majority 
response answer key. Additionally, when informants* responses were scored 
against these external standards, the proportion of correct answers was 
virtually isomorphic to competence. For the countries matching task, 
competence correlated 0.99 with proportion correct (scored according to the 
atlas) and for the fabrics matching task, competence correlated 0.97 with 
proportion correct (scored according to the fabric stores' labels). 



DISCUSSION 

This study supplies validating support for the cultural consensus model. 
Overall, each alternate indicator of knowledge correlated positively with 
cultural competence and all measures converged in measuring a single 
underlying knowledge construct. When there were objective external stan- 
dards for the answers to the consensus task questions, the consensus 
estimated answer keys produced very similar answers to the external stan- 
dards and were in greater agreement with these standards than the majority 
response answer keys were for the matching tasks. Also, the proportion 
of correct answers according to these external standards was almost 
perfectly correlated with competence. Furthermore, an unobtrusive measure 
of informants' motivation was unrelated to any indicator of knowledge. 

Of the alternate measures of informant knowledge, free listing capacity 
and self-reported recognition ability corresponded most closely with cultural 
competence. The quantity of domain information a person possesses 
(number of items s/he can free list, number of items s/he reports to be 
able to recognize) does indeed appear related to the quality of domain infor- 
mation s/he holds (as reflected by agreement with others). These findings 
are consistent with other studies. Boster (1985) observed that more com- 
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petent Aguaruna informants (i.e., those who agreed more with others) in 
a manioc identification task used a greater number of variety names than 
less competent informants (i.e., those who agreed less with others). This 
result suggests that the higher competence informants knew the names of 
more manioc types than the lower competence informants. The associa- 
tion between self-reported recognition ability and objectively measured 
knowledge was also indicated by Berdie (1971). He found that undergrad- 
uates who performed better on a multiple choice test regarding facts about 
particular famous persons (artists, authors, and public figures) reported 
knowing more about the famous persons overall than undergraduates who 
performed worse on the multiple choice test 

Moreover, the relationship between cultural competence and free listing 
capacity documented in this paper generalizes already established associ- 
ations between domain knowledge and amount of recall in episodic memory 
tasks. Cognitive psychologists have demonstrated that high domain 
knowledge experts recall more (text, lexical items, new information, etc. 
related to a particular semantic domain) than low domain knowledge novices 
in tasks where the researcher presents subjects with the material to be 
recalled (e.g., McKeithen, Reitman, Rueter, and Hirtle 1981; Morris, 
Tweedy, and Gruneberg 1985; Norman, Brooks, and Allen 1989; Schneider 
and Bjorklund 1992; Spilich, Vesonder, Chiesi, and Voss 1979). In these 
studies, individuals* knowledge levels were interred from their education 
level or occupation, or measured by tests of knowledge for which researchers 
knew the correct answers a priori. In addition, subjects in these studies were 
explicitly given time to study the material for later recall. My results show, 
however, that higher domain knowledge informants (as determined by a 
measure of knowledge that does not require researchers to know the correct 
answers a priori) recall more in a semantic memory task (where recall is 
based on material learned over a long period of time) than lower domain 
knowledge informants. 3 

In the current study, the relationship between self-rating of knowledge 
and competence tended to be modest and unreliable. In general, self-rating 
of knowledge was a poorer quality measure of knowledge than the other 
indicators, as demonstrated by the factor analyses of the inter-indicator 
correlation matrices. Self-rating of knowledge was not always consistently 
related to free listing capacity and it was less strongly associated with 
self-reported recognition ability than free listing capacity was. 

The lower quality of self-rating of knowledge as an indicator of knowl- 
edge could be due to the fact that a self-rating is based on a single response 
(or item, in psychometric terms), whereas the other measures are based 
on multiple responses (items), and therefore may contain more informa- 
tion. Self -rating of knowledge is also distinct from the other indicators of 
knowledge in that the others directly tap facets of linguistic knowledge, 
while a self-rating does not. Other possible explanations for the relative 
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weakness of self-rating of knowledge as an indicator include: informants 
making self-ratings with reference to their personal networks (with domain 
knowledge distributed unevenly across different personal networks) instead 
of college students in general, the small amount of feedback individuals 
tend to receive about their domain knowledge, the specificity of the domains, 
and/or individual differences in the use of self-rating scales. 

Indeed, one or several of these factors may account for the variable 
validity of self-ratings of other attributes. For example, self-ratings of such 
abilities as manual speed and accuracy, numerical ability, and spatial 
orientation were only modestly correlated with scores on standard tests of 
those abilities (r's between 0.05 and 0.41) in 114 undergraduates (DeNisi 
and Shaw 1977). In contrast, Schrauger and Osberg (1981) found in a review 
of many studies that the correlations between students' self -ratings of 
academic ability or self-predicted grades and students' actual grades in 
the following academic term(s) were somewhat higher, ranging between 
0.41 and 0.64. Also, Iannucci (1991) found that in a college sorority where 
informants were well-acquainted with each other, informants' self-ranks 
on several traits (physical attractiveness, assextiveness, and intelligence) 
were correlated with consensual peer rankings in the range of 0.54 to 0.66. 

There appeared to be little genuine intracultural variation among infor- 
mants for some of the triads tests and most of the ranking tasks. The 
correlations between competence and the other indicators tended to be more 
strongly positive when there was noticeable intracultural variation among 
informants on the consensus tasks than when there was not. Careful inspec- 
tion of Tables IA, IB, II A, IIB, IV, and V reveals that the strongest 
associations between cultural competence and free listing capacity, in 
particular, occurred when intracultural variation in informants* knowledge 
on the consensus task was comparatively large with respect to the simula- 
tions. The lack of appreciable intracultural variation most likely depressed 
the association between cultural competence and internal consistency of 
responses in paired comparison tasks observed in previous research. In fact* 
the two tasks in which Welter's (1984) informants displayed the smallest 
variance in competence were the same tasks in which the competence - 
consistency relationship was the weakest. Likewise, mere were two samples 
which did not display the expected relationship between competence and 
consistency in Brewer et c/.'s (1991) study. By employing the same 
simulation procedures for the ranking data described earlier (Weller 1987), 
I found that genuine intracultural variation in informants' knowledge on the 
paired comparison task in these two samples was also doubtful. 

Despite the questionable degree of intracultural variation in the ranking 
tasks, there still remained a perceptible relationship between cultural com- 
petence in triads test and ranking tasks, with a mean r of 0.29 for the 
high and low salience lists. Boster (1985) also showed that informants' 
competences in three separate tasks (general, easy manioc, hard manioc 
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plant identification) were moderately related, with r*s ranging between 
0.35 and 0.6. For the two tasks with the greatest variation in competence 
in Weller's (1984) study, informants* competences in the two tasks were 
also reasonably correlated (rho - 0.58). Hence, competence in one task in 
a domain corresponds to competence in another task in a domain. 

Boster (1985) and Weller (1984) noted that individual differences in 
knowledge can only be ascertained after analyzing inter-informant agree- 
ment. The results from the present study, however, suggest that tree listing 
capacity might be useful as a rapid and preliminary measure of infor- 
mants* knowledge in specific semantic domains. After conducting brief free 
listing interviews with a number of informants, a researcher could then pick 
a subset of informants who mentioned the most terms in the free listing task 
for further interviewing. Applied anthropologists, in particular, could benefit 
from using this quick and easy data-driven (Johnson 1990) method of 
identifying knowledgeable informants (cf. van Willigen and Final 1991): 

Furthermore, free listing capacity could be used in conjunction with 
cultural consensus analysis in determining which informants are truly 
knowledgeable. Boster and Johnson (1989) found that novice undergradu- 
ates with little fishing experience agreed more with each other on the 
similarity of line-drawn fish stimuli than did expert fishermen. Novices 
made their judgments on the basis of morphological criteria while experts 
made their judgements in terms of both functional and morphological 
criteria, resulting in their lower agreement In a case like this, experts surely 
would free list more items from the domain under study than novices, which 
would alert researchers that agreement on a structured task may be super- 
ficial and not appropriately considered a reflection of cultural knowledge. 4 
Free listing capacity also could be a useful indicator of knowledge when 
there are only a few knowledgeable informants in a sample and no overall 
consensus exists among informants in their responses to systematic inter- 
view questions. However, free listing capacity might not have universal 
applicability as an indicator of knowledge. It might not signify knowl- 
edge in domains with a small and/or fixed number of items (e.g., methods 
of AIDS transmission). In such domains, informants who free list many 
items might not be very knowledgeable. 

The results from the present study add to the growing literature on the 
distribution of cultural knowledge and characteristics of knowledgeable 
informants. Other work has shown that there are several non-cognitive indi- 
cators of knowledge, including age and length of experience with a domain 
(Boster 1985; Brewer 1992a; Garro 1986; Weller 1984), literacy (Weller 
1984), .normalcy of experience (Weller, Romney, and Orr 1985), and cen- 
trality in a social network (Brewer 1992a, 1992b). Future research should 
investigate peer assessment of knowledge and other potential indicators 
of knowledge to validate further the cultural consensus model and to refine 
procedures for selecting knowledgeable informants. Peer assessments are 



perhaps the most reliable and valid predictors of numerous individual attrib- 
utes, such as job performance (Love 1981), leadership ability (Amir, 
Kovarsky, and Sharan 1970) and personality traits (Iannucci 1991; Romney 
et al. n.d.). More work is also required in different domains and with 
informants in other cultures to generalize the findings reported here. 

NOTES 

* An earlier version of this paper was presented at die 92nd Annual Meeting of the American 
Anthropological Association in Washington, D.C. This research was supported by an Air 
Force Laboratory Graduate Fellowship awarded to the author by the Air Force Office of 
Scientific Research. I thank A. Kimball Romney and William H. Batchelder for their 
suggestions at various points during this research and Laura Yang for her assistance in data 
collection. 

1 The first sentence of free listing instructions for the countries domain was "What are all 
the countries in the world?" 

2 I modified Weller's (1987) simulation procedures for ranking data slightly by transforming 
the interval scale values produced in the simulations to ranks. 

3 The only published study on the relationship between free listing capacity and knowl- 
edge I found in my search of the literature was Hutchinson's (1983) report that pharmacy 
students (presumed experts) free listed significantly more brand names of nonprescription 
cold remedies man marketing students (presumed novices). 

4 Boster and Johnson (1990) were primarily concerned with comparing novices and experts' 
patterns of agreement, not determining which set of informants knew more about fish. This 
example merely shows that free listing capacity can be used to test whether cultural com- 
petence in a particular task is a reflection of cultural knowledge. 
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APPENDIX 
ITEM LISTS 
High salience 

Birds: pigeon, hummingbird, crow, parrot, eagle, seagull, robin, dove, btuejay, chicken, hawk, 
owl, pelican, parakeet, vulture, flamingo, sparrow, duck, ostrich, quail, turkey 

Countries': China, Japan, Italy, France, Canada, Vietnam, Switzerland, U.S.A., Spain, 
Mexico, Iran, Chile, Germany, Iraq, England, Egypt, Greece, Finland, Israel, U.S.S.R., 
Sweden 
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Diseases: A.LD.S., cancer, Alzheimer's disease, syphilis, gonorrhea, leukemia, herpes, flu, 
polio, diabetes, tuberculosis, chicken pox, measles, hepatitis, heart disease, malaria* 
smallpox, Parkinson's disease, Down's syndrome, Lou Gehrig's disease, alcoholism 

Fabrics: cotton, silk, polyester, rayon, wool, leather, nylon, suede, spandex, velvet, satin, 
canvas, taffeta, chiffon, cashmere, denim, linen, lace, flannel, acrylic, lycra 

Flowers: rose, carnation, daisy, tulip, sunflower, lily, violet, dandelion, orchid, chrysanthemum, 
daffodil, iris, poppy, lilac, bird of paradise, baby's breath, gardenia, pansy, petunia, 
magnolia, marigold 



Low salience 

Birds: condor, finch, toucan, raven, blackbird, peacock, swallow, goose, penguin, woodpecker, 
bald eagle, canary, rooster, cockatoo, swan, cockatiel, cardinal, roadnmner, crane, bluebird, 
mockingbird 

Countries': Laos, New Zealand, Haiti, North Korea, Ecuador, Zimbabwe, Indonesia, Malaysia, 

EI Salvador, Yugoslavia, Jamaica, Kuwait, Colombia, Mongolia, Romania, Peru, Jordan, 

Paraguay, Lebanon, Nicaragua, Kenya 
Diseases: multiple sclerosis, cold, Lyme disease, ulcer, tumor, gingivitis, tetanus, yeast 

infection, rabies, cataracts, mumps, cerebral palsy, Huntington's disease, emphysema, 

elephantitis, hemophilia, pneumonia, whooping cough, allergy, strep throat, schizophrenia 
Fabrics: canvas, taffeta, chiffon, cashmere, denim, linen, lace, flannel, vinyl, acrylic, lycra, 

dacron, corduroy, tweed, lame, felt, angora, plastic, seersucker, gabardine, crepe 
Flowers: poppy, lilac, bird of paradise, baby's breath, gardenia, pansy, petunia, magnolia, 

marigold, hibiscus, geranium, cherry blossom, morning glory, mum, impatiens, tiger 

lily, water lily, buttercup, sweet pea, snapdragon, azalea 



Combined lists 

Countries*: 

Map area 
World 
Asia 

Africa/Middle East 
Europe 

Norm America 
South America 

Fabrics: cotton, silk, polyester, rayon, wool, nylon, velvet, satin, canvas, taffeta, chiffon, 
denim, linen, lace, flannel, vinyl, acrylic, lycra, dacron, corduroy, tweed, lame, felt, plastic, 
seersucker, gabardine, crepe 

* The U.S.S.R. was still an intact country when the high and low salience tasks data were 
collected and Yugoslavia was still an intact country when both high and low salience and 
matching tasks data were collected. 



Countries numbered on map 
New Zealand 

Iraq, Kuwait, Iran, Mongolia, China, North Korea, Japan, Laos, 

Thailand, Vietnam, Malaysia, Indonesia 

Lebanon, Israel, Jordan, Egypt, Kenya, Zimbabwe 

Finland, Sweden, England, Germany, France, Switzerland, Spain, Italy, 

Yugoslavia, Romania, Greece 

Canada, U.S.A., Mexico, El Salvador, Nicaragua 

Colombia, Ecuador, Peru, Chile, Paraguay 
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ABSTRACT: This paper reports on the use of the pile sort technique to explore categories 
for 30 siSwati terms for acute respiratory infections. 

We interviewed 29 mothers of children under the age of 5 who were randomly selected 
from 5 urban and 12 rural sites throughout Swaziland, as well as the 17 health providers 
and 13 traditional healers whom mothers reported they would contact if their child became 
ill. 

Results of our study suggest that there are siSwati illness terms mat correspond to many 
of the signs and symptoms of upper and lower acute respiratory infection. The respondents 
differentiated these terms into at least two distinct groups: one group included terms asso- 
ciated with the symptoms of common colds or flu that mothers manage at home, while the 
other group included terms that refer to more serious illnesses for which individual care- 
takers usually seek further professional treatment However, the "cognitive boundary** between 
these two groups is not always clear. We expected differences in the folk and biomedical 
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